Class-Incremental Learning based on Label Generation

Despite the great success of pre-trained language models, it is still a challenge to use these models for continual learning, especially for the class-incremental learning (CIL) setting due to catastrophic forgetting (CF). This paper reports our finding that if we formulate CIL as a continual label generation problem, CF is drastically reduced and the generalizable representations of pre-trained models can be better retained. We thus propose a new CIL method (VAG) that also leverages the sparsity of vocabulary to focus the generation and creates pseudo-replay samples by using label semantics. Experimental results show that VAG outperforms baselines by a large margin.Comment: 12 pages, ACL 2023 Main Conferenc

Adapting a Language Model While Preserving its General Knowledge

Domain-adaptive pre-training (or DA-training for short), also known as post-training, aims to train a pre-trained general-purpose language model (LM) using an unlabeled corpus of a particular domain to adapt the LM so that end-tasks in the domain can give improved performances. However, existing DA-training methods are in some sense blind as they do not explicitly identify what knowledge in the LM should be preserved and what should be changed by the domain corpus. This paper shows that the existing methods are suboptimal and proposes a novel method to perform a more informed adaptation of the knowledge in the LM by (1) soft-masking the attention heads based on their importance to best preserve the general knowledge in the LM and (2) contrasting the representations of the general and the full (both general and domain knowledge) to learn an integrated representation with both general and domain-specific knowledge. Experimental results will demonstrate the effectiveness of the proposed approach.Comment: EMNLP 202

Inferring Tabular Analysis Metadata by Infusing Distribution and Knowledge Information

Many data analysis tasks heavily rely on a deep understanding of tables (multi-dimensional data). Across the tasks, there exist comonly used metadata attributes of table fields / columns. In this paper, we identify four such analysis metadata: Measure/dimension dichotomy, common field roles, semantic field type, and default aggregation function. While those metadata face challenges of insufficient supervision signals, utilizing existing knowledge and understanding distribution. To inference these metadata for a raw table, we propose our multi-tasking Metadata model which fuses field distribution and knowledge graph information into pre-trained tabular models. For model training and evaluation, we collect a large corpus (~582k tables from private spreadsheet and public tabular datasets) of analysis metadata by using diverse smart supervisions from downstream tasks. Our best model has accuracy = 98%, hit rate at top-1 > 67%, accuracy > 80%, and accuracy = 88% for the four analysis metadata inference tasks, respectively. It outperforms a series of baselines that are based on rules, traditional machine learning methods, and pre-trained tabular models. Analysis metadata models are deployed in a popular data analysis product, helping downstream intelligent features such as insights mining, chart / pivot table recommendation, and natural language QA...Comment: 13pages, 7 figures, 9 table

Genome-wide association study (GWAS) datasets used in the study.

Genome-wide association study (GWAS) datasets used in the study.</p

S3 File -

BackgroundAn association between primary biliary cholangitis (PBC) and connective tissue diseases (CTDs) [rheumatoid arthritis (RA), systemic lupus erythematosus (SLE), Sjögren’s syndrome (SS), systemic sclerosis (SSc)] has been found in observational studies. However, the direction causality is unclear. The aim of this study was to assess the causality between PBC and CTDs and to promote early screening, pre-emptive therapy, and accurate stratification.MethodsA two-sample Mendelian randomization (MR) analysis was performed to assess the causal relationship between PBC [Genome-Wide Association Study (GWAS) meta-analysis, 8021 cases/16498 controls], and SLE (GWAS meta-analysis, 8021 cases/16489 controls), RA(FinnGen, 6236 cases/14727 controls), SS(FinnGen, 2495 cases/365533 controls), SSc (FinnGen, 302 cases/213145 controls). Inverse variance weighting (IVW) was used as the primary analysis method, supplemented by four sensitivity analyses to assess the robustness of the results.ResultsThe IVW revealed that genetically predicted PBC increased the risk of SLE [odd’s ratio (OR) = 1.43, 95% confidence interval (CI) 1.30–1.58, P ConclusionsOur study provided new genetic evidence for a causal relationship between PBC and CTDs. PBC increased the risk of SLE, RA, and SS. Our findings highlighted the importance of active screening and intervention for CTDs in patients with PBC.</div

S1 File -

BackgroundAn association between primary biliary cholangitis (PBC) and connective tissue diseases (CTDs) [rheumatoid arthritis (RA), systemic lupus erythematosus (SLE), Sjögren’s syndrome (SS), systemic sclerosis (SSc)] has been found in observational studies. However, the direction causality is unclear. The aim of this study was to assess the causality between PBC and CTDs and to promote early screening, pre-emptive therapy, and accurate stratification.MethodsA two-sample Mendelian randomization (MR) analysis was performed to assess the causal relationship between PBC [Genome-Wide Association Study (GWAS) meta-analysis, 8021 cases/16498 controls], and SLE (GWAS meta-analysis, 8021 cases/16489 controls), RA(FinnGen, 6236 cases/14727 controls), SS(FinnGen, 2495 cases/365533 controls), SSc (FinnGen, 302 cases/213145 controls). Inverse variance weighting (IVW) was used as the primary analysis method, supplemented by four sensitivity analyses to assess the robustness of the results.ResultsThe IVW revealed that genetically predicted PBC increased the risk of SLE [odd’s ratio (OR) = 1.43, 95% confidence interval (CI) 1.30–1.58, P ConclusionsOur study provided new genetic evidence for a causal relationship between PBC and CTDs. PBC increased the risk of SLE, RA, and SS. Our findings highlighted the importance of active screening and intervention for CTDs in patients with PBC.</div

Diagram of the two-sample Mendelian randomization (MR) study for the association between PBC and CTDs.

PBC, primary biliary cholangitis; SLE, systemic lupus erythematosus; RA, rheumatoid arthritis; SS, Sjögren’s syndrome; SSc, systemic sclerosis.</p

Causal effects of CTDs on the risk of PBC assessed by inverse Mendelian randomization method.

CTDs for exposure, PBC for outcome. PBC, primary biliary cholangitis; SLE, systemic lupus erythematosus; RA, rheumatoid arthritis; SS, Sjögren’s syndrome; SSc, systemic sclerosis. SNP, single-nucleotide polymorphism; OR, odds ratio; CI, confidence interval. IVW, inverse-variance weighted; IVW(mre), inverse-variance weighted(multiplicative random effects).</p

Causal effects of PBC on the risk of CTDs assessed by different Mendelian randomization methods.

PBC for exposure, CTDs for outcome. PBC, primary biliary cholangitis; SLE, systemic lupus erythematosus; RA, rheumatoid arthritis; SS, Sjögren’s syndrome; SSc, systemic sclerosis. SNP, single-nucleotide polymorphism; OR, odds ratio; CI, confidence interval. IVW, inverse-variance weighted; IVW(mre), inverse-variance weighted(multiplicative random effects).</p